dbt CoreでローカルからBigQueryを操作するためのセットアップ方法

brew update
brew install git
brew install pyenv
brew install pyenv-virtualenv
brew install --cask google-cloud-sdk

# 作業ディレクトリを作成
cd
mkdir dbt_test && cd dbt_test

# 仮想環境の準備
pyenv install 3.12.0
pyenv virtualenv 3.12.0 python312
pyenv local python312

# 確認
pyenv versions
python --version

# requirements.txtに書き込む
echo "dbt-bigquery==1.8.2" > requirements.txt
echo "dbt-core==1.8.3" >> requirements.txt

# 必要なライブラリをインストール
pip install -r requirements.txt

# 確認
dbt --version

Core:
  - installed: 1.8.3
  - latest:    1.8.3 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

gcloud auth application-default login

dbt init test_project

Running with dbt=1.8.3

Your new dbt project "test_project" was created!

For more information on how to configure the profiles.yml file,
please consult the dbt documentation here:

  https://docs.getdbt.com/docs/configure-your-profile

One more thing:

Need help? Don't hesitate to reach out to us via GitHub issues or on Slack:

  https://community.getdbt.com/

Happy modeling!

Setting up your profile.

Which database would you like to use?
[1] bigquery

(Don't see the one you want? https://docs.getdbt.com/docs/available-adapters)

Enter a number: 1

[1] oauth
[2] service_account
Desired authentication method option (enter a number): 1

project (GCP project id): [プロジェクトID]
dataset (the name of your dbt dataset): dbt_dataset
threads (1 or more): 1
job_execution_timeout_seconds [300]: 
[1] US
[2] EU
Desired location option (enter a number): 1

Profile test_project written to /Users/[ユーザーディレクトリ]/.dbt/profiles.yml using target's profile_template.yml and your supplied values. Run 'dbt debug' to validate the connection.

cd test_project
dbt debug

Running with dbt=1.8.3
dbt version: 1.8.3
python version: 3.12.0
python path: /Users/[ユーザーディレクトリ]/.pyenv/versions/3.12.0/envs/python312/bin/python3.12
os info: macOS-13.5.2-arm64-arm-64bit
Using profiles dir at /Users/[ユーザーディレクトリ]/.dbt
Using profiles.yml file at /Users/[ユーザーディレクトリ]/.dbt/profiles.yml
Using dbt_project.yml file at /Users/[ユーザーディレクトリ]/python3_venv/test_project/dbt_project.yml
adapter type: bigquery
adapter version: 1.8.2
Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]
Required dependencies:
  - git [OK found]

Connection:
  method: oauth
  database: [プロジェクトID]
  execution_project: [プロジェクトID]
  schema: dbt_dataset
  location: US
  priority: interactive
  maximum_bytes_billed: None
  impersonate_service_account: None
  job_retry_deadline_seconds: None
  job_retries: 1
  job_creation_timeout_seconds: None
  job_execution_timeout_seconds: 300
  timeout_seconds: 300
  client_id: None
  token_uri: None
  dataproc_region: None
  dataproc_cluster_name: None
  gcs_bucket: None
  dataproc_batch: None
Registered adapter: bigquery=1.8.2
  Connection test: [OK connection ok]

All checks passed!

dbt run

Running with dbt=1.8.3
Registered adapter: bigquery=1.8.2
Unable to do partial parsing because saved manifest not found. Starting full parse.
Found 2 models, 4 data tests, 471 macros

Concurrency: 1 threads (target='dev')

1 of 2 START sql table model dbt_dataset.my_first_dbt_model .................... [RUN]
1 of 2 OK created sql table model dbt_dataset.my_first_dbt_model ............... [CREATE TABLE (2.0 rows, 0 processed) in 3.46s]
2 of 2 START sql view model dbt_dataset.my_second_dbt_model .................... [RUN]
2 of 2 OK created sql view model dbt_dataset.my_second_dbt_model ............... [CREATE VIEW (0 processed) in 1.60s]

Finished running 1 table model, 1 view model in 0 hours 0 minutes and 6.97 seconds (6.97s).
Completed successfully

Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2